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An orthodox approach to content-addressable memory 


note that this is a brute-force decoder 
for the error-correcting code whose codewords are the required memories 
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Neural Networks, Part Il: 


Feedback Networks 


The Hopfield Network 


Chapter 42 
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Architecture 
Feedback network 
Symmetric weights wij = ши, wii 0 
Activity rule 
Continuous Hopfield network: John Hopfield 
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Content-addressable memory challenge 


@Make a dynamical system 
925 dynamical variables, 300 parameters (4-bit precision) 
that has attracting fixed points at desired memories 
such that noisy versions are automatically cleaned up 
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“апа such that new memories сап be added incrementally 
@AND robust to corruption of more than half the parameters 


Hopfield network and Hebbian learning Chapter 42 


Architecture 
Feedback network 
Symmetric weights wij = wj; Wi = 0 
Activity rule 
Continuous Hopfield network: 
“= >” WijTj + 0; 
j 
ті = tanha; 
Binary Hopfield network: 
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Learning rule 
Hebb rule 
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McGurk effect 


_ From Wikipedia, the free encyclopedia 


The McGurk effect is а р ‘ual phenomenon that demonstrates an Interaction b 
perception. The Illusion occurs when the auditory component of one sound 15 paired 
another sound, leading to the perception of a third sound.!!! The visual Information = 
speak changes the way they hear the sound.!*! People who are used to watching dub 
Who are not susceptible to the McGurk effect because they have, to some extent, (ез 
_ аге getting from the mouths of the "speakers".'?! If a person 15 getting poor quality а 

Visual Information, they may be more likely to experience the McGurk етеск "| Intec 

Information may also Influence whether a person will experience the effect. People t 
__ һауе been shown to be more susceptible to the effect.') Many people are affected d 
| based on many factors, brain damages or disorders. 


1 Background | 
2 Brain influences 
3 Factors 

4 Other languages 

5 Hearing impairment 


Hebbian learning 


Desired memories Weights 


EIE 6 
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The Hopfield network 


(Why does Hopfield net have stable states at all? 
(What is the capacity of the Hopfield net? 
(What else can we do with them? 


О: Where have we seen the Hopfield net before? 


Wij = wj, Wi =0 
Dynamics Hint: 


= > ів ат = У Лап + ћу 
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ті = tanha; То = tanh an 


Variational methods іі 


When we approximated the spin system whose energy function was 
1 
E(xJ) = -5 тзт» - > һа 


with a separable distribution 


MERA 


and optimized Q so as to minimize the variational free energy 


BP (а) 23 Q(x; a) E(x; J) - > Qlx;a) n= ae T 


we found that the iterative equations 


аһ = В (zs. р 2 and т, = tanh(an) 


were guaranteed to decrease the variational free energy 
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Capacity of Hopfield network ere 


Overlap between a desired memory and the nearest stable state 
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N/I N/I 
Capacity for random patterns: Nerit = 0.1387 


0.24 bits per weight. 
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The Hopfield network 


(Why does Hopfield net have stable states at all? 
What is the capacity of the Hopfield net? 
(What else can we do with them? 
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Hopfield networks for optimization ез 


"The Travelling Scholar Problem" (Sree Aiyer, 1991) 
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Q: What is this? 


1 £ dis ? 
P(x) = + .Bleızar325 + 22232416 + 71237477] 
Zo 
where x = (21,22,23,...,27), Un € £1 
and 3 = large 


& What is this? 

N 
B [21222325 + 22237426 + 21732427] + PS b Ln 
Pı(x) = е n=1 
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Idea 


N 
В [а 22325 + тотзтате + 71737477] + » бота 
Р, = 25, n=1 
1 (x) Zi e 
@Solve the decoding problem 
max Р. (x) 
x 


using approximate methods? 


eg, Approximate Pi(x) by Q(x) 
using a variational method 


Electronics Letters, March 1995 


Free energy minimisation algorithm for 
decoding and cryptanalysis 


D.J.C. MacKay 


Indexing terms: Decoding, Cryptography 


An algorithm is derived for inferring a binary vector s given nois 
observations of As modulo 2, where A is a binary matrix. Th 
binary vector is replaced by a vector of probabilities, optimised b 
free energy minimisation. Experiments on the inference of th 
state of a linear feedback shift register indicate that this algorithı 
supersedes the Meier and Staffelbach polynomial algorithm. 


аа NO Шыр Ye Wa? 


Electronics Letters, August 1996 


Near Shannon limit performance of low 
density parity check codes 


D.J.C. MacKay and R.M. Neal 
[Low Density Parity Check Codes: Gallager 1962] 


Indexing terms: Probabilistic decoding, Error correction codes 


The authors report the empirical performance of Gallager’s lo 
density parity check codes on Gaussian channels. They show th: 
performance substantially better than that of standar 
convolutional and concatenated codes can be achieved; indeed th 
performance is almost as close to the Shannon limit as that ‹ 
turbo codes. 


Low Density Parity Check Code Gna? 


TRANSMITTED: 
We demonstrate a large code that en- 


codes K = 10000 source bits into 

N = 20000 transmitted bits. 

Each parity bit depends on about 

5000 source bits. 

The encoder is derived from a very 

sparse 10000 Х 20000 matrix Н with parity bits 
three 1s per column. 


Low Density Parity Check Code (f = 7.5%) 


Iterative probabilistic decoding 


low-density 
parity-check code 


Shannon limit, 


Probability of decoder error 


0 02 04 06 08 1 
Rate 


International Symposium on Information Theory 1997 


One paper on low-density parity-check codes 


6 sessions on LDPC codes; 10 on sparse-graph codes 
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Convolutional Codes. 
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